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(54) Automatic electronic replacement of billboards in a video image 

(57) Apparatus for automatic electronic replacement of a billboard in a video image including an automatic 
camera orientation measurement apparatus including motion measurement means operative to measure the 
Field of View (FOV) of the TV camera relative to a known reference position. The billboard preferably 
comprises a patterned chroma-key panel and the motion measurement includes measurement of camera pan, 
tilt, zoom and focus relative to known reference positions. 
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At least one drawing originally filed was informal and the print reproduced here is taken from a later filed formal copy. 
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1. setup rotation matrix 

cos(P s ) 0 -sin(Ps)iri 0 0 
Rr 0 1 0 0 cos(T s ) sin(T s ) 
sin(Ps) 0 cos(P s )J|0-sin(T s ) cos(T s ) 
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2. prediction rotation matrix 



(cos(P p ) 0 -sin(P p ) 

0 1 0 
|sin(IV) 0 cos(P P ) 



1 0 0 

0 cos(T p ) sin(T p ) 
0-sin(T p ) cos(T p ) 
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3. setup to prediction perspective transformation 
Rsp =Rs ^Rp 



4. setup to prediction point transformation 



Up = 
Vp = 



Rsp[0H01us+ RspHH0|vs + Rsp|2H01 
Rsp|0]I21u $ + Rspllimvs + Rsp |21|21 

Rsp[01Hlus+ Rs P |l||l]v s + Rg |2J|lj 
Rsp |0||2]us + Rp Il)|2|* + Rsp (2|(2) 
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5. image plane to frame-buffer 

x = (u-A„(u,v))M x +Xc 
y = (v-A v (u,v))M Y +Y c 
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6. aberrations 

A 0 (u,v) = (g 1 + g 3 )u 2 +g 4 uv + g 1 v 2 + k, u(u 2 +v 2 ) 
Av(u,v) = g 2 u 2 +g 3 uv +(g 2 + g 4 )v 2 +k 1 v(u 2 + v2) 
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METHOD AND APPARATUS FOR AUTOMATIC ELECTRONIC 
REPLACEMENT OF BILLBOARDS IN A VIDEO IMAGE 

The present invention relates to a method and apparatus for 
5 automatically replacing billboards in a video image. 

The present invention has particular use in electronic replacement 
of billboards in a stadium or other venue but can be used to provide 
accurate data relating to camera orientation for other purposes. 

10 

In previous systems it has been proposed to electronically replace 
billboards in a stadium which are viewed by a viewer on television. The 
billboards in the stadium are televised by a TV camera and the boards are 
electronically altered so that the TV viewer at home sees a different board 
IS to the spectator in the stadium or other venue. 

The known systems such as those described in US 5,266,933, an 
apparatus and method for electronically altering video images is disclosed. 
The apparatus and method disclosed in the US patent and also in US 
20 patent 5,353,392 whilst theoretically allowing replacement of billboards 
do not solve the many practical problems encountered in real 
environments. Most of these problems are related to the recognition and 
replacement processes. 

25 Relying entirely on pattern recognition techniques which utilize only 

the video signal to identify and localise billboards for replacement 
introduces major problems which affect the practical value of such a 
system. 

30 Clearly, any pattern recognition scheme, including those described 

1 



in US 5,264,933 and US 5,353,392 must rely on useful visible features in 
the image that can be compared with pre-defined descriptions. Such 
features should be located inside the billboard or at its neighbourhood. 

5 In realistic situations, the visibility of these features might change, 

continuously or otherwise from practically zero to a some threshold 
visibility which allows the pattern recognition scheme to work properly. 
These changes can occur in the direction of growing or reducing visibility. 



10 Such situations include : 

• Acceleration or de-acceleration of camera motion 
introducing a huge amount of blur. 

• Excessive zooming-in or zooming-out of the billboard. 

• Excessive occlusion by players. 

15 • Entering or exiting a camera's field of view by any 

combination of pan, tilt and zoom operations. 

• Any combination of the above mentioned mechanisms. 



Therefore, in practical situations, a continuous replacement of 
20 billboards, is not possible. Even if an interrupted replacement was 
allowed, it would require a delay of at least a few seconds to decide 
whether the resulting replacement interval is acceptable or not. Such a 
delay is usually not permitted in live broadcasting of sports events. 

25 Replacing arbitrary billboards introduces further problems. A 

seamless replacement requires to identify the foreground objects occluding 
the billboard in order to inhibit replacement at places of occlusion. 
Foreground objects mainly consist of players but also the ball or other 
objects. Consider now a player with a red shirt, occluding a part of a 

30 similarly red portion of a billboard. Colour contrast cannot be used 
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robustly to identify occlusion. Furthermore, since the player is a non- 
rigid object, motion or shape information cannot be used accurately 
enough to guarantee perfect replacement. 

5 Another problem which may arise in practical situation is resolution 

of billboard identity. Consider two identical billboards positioned at two 
different locations in the arena. Suppose different replacement billboards 
are assigned to each of these physical billboards, then one must be able to 
tell which one is which. This can prove to be extremely difficult 

10 especially if no unambiguous features are visible. 

This invention describes a robust system for billboard replacement, 
based on some or all of the following key-elements: 



IS • Pan, tilt, zoom and focus sensors attached to the camera, 

which enable after a proper set up procedure to estimate the 
presence and location of billboards in any given video field. 

• Image processing methods and their embodiment which 
enable to refine the sensors' estimates. 

20 • Physical billboards which are coloured properly to enable 

the efficient detection of occlusion by chroma-key 
techniques. 

• Colour variation or a pattern within the physical billboard 
for further enhancing the performance of the image 

25 processing methods. 



The present invention has a first object to provide a method and 
apparatus which enables identification of the location of a billboard or 
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other static object in a stadium or other venue in any weather conditions 
with any panning speed of the camera and with any other change in 
camera parameters. 

5 The present invention therefore provides apparatus for automatic 

electronic replacement of a billboard in a video image including an 
automatic camera orientation measurement apparatus including motion 
measurement means operative to measure the Held of View (FOV) of the 
TV camera relative to a known reference position. 

10 

The present invention also preferably provides apparatus for 
automatic electronic replacement of a billboard in a video image, including 
image processing means for processing video signals generated by the TV 
camera, in which said processing means includes calibration means for 

15 periodically automatically calibrating the motion measurement means, 
apparatus in which the motion measurement means includes means for 
measuring the pan tilt, zoom or focus of the camera relative to known 
reference positions and apparatus in which the motion measurement means 
includes means for measuring the pan tilt, zoom and focus of the camera 

20 relative to known reference positions. 

The present invention therefore uses dynamic recalibration to 
correct for residual sensor errors or abberations in an imperfect model ami 
for sensor drift over time. Thus it is possible in accordance with the 
25 present invention to use less stable sensors and the apparatus and method 
in accordance with the present invention can accommodate movement in 
the camera position. The image correction process for calibration of the 
sensors eliminates the necessity to keep the sensors stable by mechanical 
means by recalibration automatically with reference to the video image. 

30 
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In the initial set up procedure corrections can be incorporated for 
calibration for billboards which are, for example, not in the centre of the 
Field of View - for example a billboard which is in the top left-hand 
corner of the screen can be adjusted by, for example, 3 pixels to take into 
5 account abberations in the camera. 

Further problems which arise in the prior art systems are firstly 
when the billboard is either substantially totally occluded or secondly, is 
occluded by an object, such as a player, of the same colour as die real 
10 sign on the billboard. 

This can firstly, as explained above, lead to non-recognition of the 
billboard and also secondly, to difficulty in satisfactory replacement of 
the billboard. 

15 

In the first case the real billboard can have already been replaced 
in the video image but if the camera zooms into a close up or if a different 
camera is used for the dose up then the lock may be lost due to only a 
very small portion of the billboard being in view. In the second case the 

20 player may have on a strip which is the same colour as the billboard. The 
prior art systems propose to distinguish the billboard from the player on 
the basis of movement if the colours are the same and to analyse the 
"moving" pixels to determine occlusion. This is reasonable in theory but 
fails in practice since not all payers are moving at all times. Thus, if a 

25 number of players move in front of a billboard and one player remains 
after the others have moved on, the electronics will not be able to 
distinguish on movement grounds. Since colours are distorted by 
floodlighting, shadows, differences in reflectivity and different lighting 
conditions for a foreground player and a background billboard, there will 

30 be occasions in practice where the system fails. In such cases either the 

5 



original billboard may re-appear on the video image or the replacement 
billboard will not be accurately occluded. 

It is again possible to introduce a delay in the video transmission 
5 to enable the electronic signal processing to be more accurate but this does 
not solve the practical problem whore a plurality of players move in 
different directions to occlude a billboard. The necessary delay is 
considered unacceptable and will in any case not solve all of the above 
problems. 

10 

In accordance with a preferred embodiment of the present invention 
it is proposed to replace die real billboards with chroma-key panels or 
with delineated areas forming chroma-key panels. 

IS Chroma-key is essentially an occlusion technique allowing, for 

example, a news reader to stand and move about in front of a chroma-key 
board, usually coloured blue or another suitable colour. The news reader 
(foreground) is distinguished from the chroma-key board (background) by 
colour differentiation and can thus move in front of the replacement 

20 background with normal occlusion of the foreground and background. 
This technique is very well known in television studio systems and is 
described in numerous US patents, including US 2,974,190 and 
4,200,980. 

25 Recently, several systems which combine camera sensors with 

chroma-key for the purpose of coordinating the movement of the graphics 
backgrounds with those of the camera, have been described and 
demonstrated. 
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[Ref. K. Haseba et al., Real-timing compositing system of a real 

6 



camera image and a computer graphic image, International Broadcasting 
Convention, 16-20 Sep. 1994, Conference publication No. 397, DEE 1994, 
pp. 656-660]. 

5 In principle, such an arrangement could be used for billboard 

replacement where the sensors unambiguously solve the recognition 
problem and the chroma-key billboard helps to handle occlusion properly. 
However due to some major differences, this arrangement should be 
enhanced. These enhancements are the basis of the present invention. 

10 

In a virtual set application, die camera is typically 2-10 metres 
away from the foreground and the entire field of view is usually replaced. 
In comparison, a billboard may be several hundred metres from the 
camera and therefore a replacement system using sensors is much more 
IS susceptible to sensor errors: 

• Due to the large focal distances, the same sensor accuracy 
will translate to larger geometric registration errors. 
Consider a rotary encoder of 81000 pluses/revolution, then 
20 the angular precision is 0.0044 degrees or 75 micro-radians. 

The repeatability is twice as bad. Consider a shooting range 
of 100 m with a field of view of 4 metres, then the FOV is 
40 milli-radian. The error translates to 768 *? 150 / 20000 
= 2.88 pixels. 

25 • Since the field of view includes many stationary objects 

(including billboards) which are not replaced, the human 
observer will be much more sensitive to the registration 
errors. Additional errors may originate from lens distortion, 
rotation axis which does not pass through the focal point, 

30 non-zero roll angle, etc. 

7 



Chroma-key is basically a technique for studios where die 
illumination is carefully designed and controlled and the controls of the 
chroma-keyer are carefully adjusted for the specific arrangement of blue- 
screen colour and illumination. 

5 

In a sports event, die conditions may be highly non-ideal and 
require some modification to the chroma-key algorithms. In particular, 
the keyer parameters should be adapted to the specific billboards being 
replaced due to changes in illumination across the arena. 

10 

Accordingly in the present invention it is proposed to use chroma- 
key panels and to replace these in the video image by the replacement 
billboards. 

IS Since it is necessary for perfect occlusion that the players or other 

occluding objects are of different colour to the chroma-key panels, it is 
proposed in a further preferred embodiment to provide chroma-key panels 
in which the colour of the panel may be changed, for example, by using 
a rotating billboard structure which is known in the art. One side could, 

20 for example, be blue and another green. Green may be preferable in a 
sports environment since players tend not to wear green as this would not 
contrast with the background sports surface. 

In a further preferred embodiment and in particular where a 
25 plurality of billboards require replacement, a patterned chroma-key board 
is used. The pattern may be of any suitable shape but is preferably 
selected to be suitable for the size and shape of the billboard or series of 
billboards and also to the anticipated video conditions. Thus if a billboard 
is only able to be viewed from a long distance then a different pattern will 
30 be selected to a billboard which is to be viewed in close up. 

8 



The pattern may comprise different colours or may be different 
shades of the same colour. The pattern may comprise vertical and 
horizontal lines or may comprise a decorative pattern, a discernible 
advertisement, company logo or other suitable wording which may be 
5 more aesthetically acceptable. 

The use of a pattern allows further discrimination of the position of 
die camera and may allow movement of the camera from a fixed position. 

10 The camera orientation data can be transmitted together with the 

video signal and will identify the position of the billboard in any weather, 
lighting or occlusion conditions. No reference is required to any feature 
within the sports venue to identify the position of the billboard. 

15 The camera sensors can be accurate to a few pixels or in physical 

terms to approximately 1 cm at a range of about 100 metres thereby 
enabling accurate replacement of any billboard. The recalibration can be 
carried out continuously or only periodically, particularly if an initial 
adjustment of the calibration of billboards not in the centre of the FOV is 

20 recorded on set up. 

By use of the chroma-key techniques there is no requirement to 
transmit any occlusion data since this can be readily inserted at a receiver 
and the occlusion inserted in the normal manner. 

25 

In a preferred arrangement within a stadium or other sports venue 
real billboards with normal advertising material will be situated on one 
side of the stadium to be viewed by a first plurality of cameras and 
chroma-key billboards will be situated on another or the opposite side to 

30 
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be viewed by a second plurality of cameras. This, for example, the home 
nation may view the normal billboards, with the international TV audience 
seeing only substituted boards. 

The present invention also provides a method for electronically 
replacing a billboard in a video image display, generated by a camera, 
comprising the steps of : 

a. identification of the position of a rectangular billboard in a 
stadium or other venue, said identification step comprising 
specifying on the video display the billboard to be replaced 
by identification of its four corners at a first camera 
position; 

b. storing the identification information; 

c. monitoring the movement of the camera in pan, tilt and 
zoom; 

d. storing the monitored movement of the camera on a field by 
field basis; and 

e. analysing the size and position of the billboard to be 
replaced from the information recorded in its first known 
position and the stored movements of the camera to provide 
information relating to the size, perspective and position of 
the billboard in the present video field; 

f. storing in a billboard replacement store a replacement 
billboard to be used in replacement of the billboard in the 
stadium; 

g. electronically altering the size and perspective of the 
replacement billboard in accordance with the camera motion 
information to conform to the size and perspective of the 
billboard to be replaced in the present video frame; and 

h. electronically replacing the billboard in the present video 

10 



frame by the replacement billboard. 

In a preferred embodiment the step of analysing the size and 
position of the billboard to be replaced comprises a further step of 
5 analysing a plurality of video scan lines to provide fine adjustment 
information relating to the exact size, perspective and position of the 
billboard to be replaced. 

In a further preferred embodiment the billboard to be replaced is 
10 blank and is of colour suitable for chroma-key replacement. Such colour 
may be of a blue or green shade, due to the fact that these colours are 
rarely found in human skin and hair. 

In a still further preferred embodiment the chroma-key billboard is 
IS patterned with a pattern of a suitable shape for the purpose of facilitating 
the fine adjustment process mentioned above. The step of analysis of the 
size and position of the billboard comprises the analysis of the pattern to 
ascertain the exact position of the billboard. 

20 In a further embodiment the correction of the sensor-based 

prediction by the analysis of the pattern will be controlled by a figure of 
merit (accuracy estimate) for the analysis, which will be computed 
automatically. 

25 In a further embodiment the step of electronically replacing the 

billboard in the present video field by the replacement billboard include 
the step of superimposing occluding objects by use of the chroma-key 
techniques. 
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In a further preferred embodiment the billboard to be replaced can 

11 



be changed to best match die colours and shades of colours on the players 
costumes, for the purpose of providing a good contrast between the 
billboards and the players. For example, if these costumes contain shades 
of blue, then a green billboard may be selected. 

5 

The backing colour can be selected between blue, green and red. 
In order for die chroma-keyer to calculate all parameters necessary to 
perform proper image compositing, the system requires a sample of the 
background colour as reference. This step can be done automatically by 
10 scanning the image and detecting the purest and brightest colour. 
Advanced chroma-keyers enable the user to manually select the area to be 
sampled. 

In a further preferred embodiment the chroma-key apparatus will 
15 have a multiplicity of set up conditions, each corresponding to a different 
region of the stadium. The camera pan, tilt and zoom information will 
allow to load the corresponding set up conditions. 

In a further embodiment, the fine adjustment information will be 
20 used to compensate drift errors of the sensors. In a practical situations, 
the sensor error will have a significant portion which is at temporal 
frequencies which are much lower than the video field rate. Thus these 
sensor-induced errors can be reliably estimated from good video field and 
subtracted from subsequent measurement. 

25 

The present invention also provides apparatus for carrying out the 
method of electronically replacing the billboard as specified hereinbefore. 

Embodiments of the present invention will now be described, by 
30 way of example with reference to the accompanying drawings in which 

12 



Figure 1 shows a stadium or other venue illustrating the apparatus 
according to the present invention; 

Figure 2 shows the video image of the stadium as seen by the 
camera in a first position; 
5 Figure 3 illustrates a stadium with billboards in several different 

positions; 

Figure 4 illustrates a zoomed camera shot of a billboard illustrating 
the problem with prior art systems; 

Figure 5 shows a patterned chroma-key billboard for use with the 
10 present invention; 

Figure 6 shows in block diagrammatic form circuitry associated 
with the camera arrangement of Figure 1 for transmitting video data and 
camera orientation data; 

Figure 7 shows in block diagrammatic form receiver circuitry for 
15 co-operation with the transmitter circuitry of Figure 6; 

Figure 8 shows a flow diagram for the operation of the circuitry of 
Figure 7; 

Figure 9 shows an arrangement for a billboard setup data store; 
Figure 10 shows an arrangement for billboard setup data; 
20 Figure 11 shows a flow diagram for perspective transformation 

computation; 

Figure 12 shows an arrangement for camera intrinsic parameters 

store; 

Figure 13 shows equations for dynamic recalibration; 
25 Figure 14 shows a flow diagram for dynamic recalibration; and 

Figure 15 shows the process of recalibration. 
With reference now to Figures 1 to 4 the principle of the present 
invention is now explained. 

30 In a stadium or other venue 10 billboards 14,16,18 are installed at 

13 



the side of a pitch represented by markings 12. These billboards are 
viewable by a camera 20. Billboards 15,17,19 may be present on the 
opposite side of the stadium for viewing by a further camera 21. TTie 
stadium terraces/seating are shown diagrammatically by the lines 11. 

5 

Camera 21 may in a preferred example be a normal TV video 
camera and will transmit its output video signal directly to a first feed 
which may serve the local population. Although we refer to camera 20 
or 21, it may be dearly understood that there could be a plurality of 
10 cameras on each side of the stadium providing differing views. 

Camera 21 in a preferred embodiment will televise boards 15,17,19 
which will be transmitted to the local population in an unchanged manner. 

15 Camera 20 will in this preferred embodiment transmit a feed to an 

international audience. Camera 20 is equipped with orientation sensing 
means which preferably comprises one or more of the following : 
pan measurement means 24; 
tilt measurement means 25; 
20 zoom measurement means 26; and 

focus measurement means 28. 

Suitable sensors may comprise the Virtual Reality Encoder from 
RADAMEC EPO, Bridge Road, Chertsey, Surrey KT16, 8U, England. 

25 

Dependent on the allowed mobility of the camera only one, several 
or all of these may be required. For example, if camera 20 is fixed in 
pan and tilt and focus and can only zoom, as in the case of some remotely 
controlled unmanned cameras then only the zoom parameter need be 
30 measured. 

14 



Most cameras in sports stadiums can zoom, tilt and pan and it is 
assumed that these parameters are measured for each camera as now 
explained. The focus is assumed to be fixed but in similar manner the 
parameter could be added if required. 

5 

Figure 2 shows the video image as seen by a viewer and in 
particular by the operator of the equipment. The camera 20 is zoomed, 
panned and/or tilted to "centre" die billboard 14 in a suitable position and 
at a reasonable size* With reference to Figure 7 each billboard is then 
10 viewed at a receiver and its position is marked preferably by using a touch 
screen 700, or keyboard mouse 702 and marking the four corners. The 
positions are stored in a store 704. 

For billboards higher in the stadium such as 30 (Figure 3) a 
IS correction factor for the camera may be stored dependent on the tilt 
position of the camera. 

Each billboard position is stored in store 704 together with the 
camera parameter information at the reference position for the camera 20 
20 obtained from the camera parameter information which is correct at the 
time that the billboard position is stored. 

The following procedure is preferably repeated for each of the 
cameras and for each of the target billboard : 

25 

1 . Point the camera at the target to obtain a stable unoccluded view of 
the target. Adjust the zoom to get a large view of the target yet 
keeping the whole target within the field of view. 

2. While the camera is not moving trigger an acquisition device, to 
30 grab a picture of the target, as well as the corresponding readings 

15 



of the sensors. 

3. Mark the corners of the target, on the video image. 

Preferably a corner detector is used to pinpoint the corners of the 
5 target at sub-pixel precision. 

This camera parameter information is obtained (Figure 6) from the 
sensors mounted on the camera and the camera movement is referenced 
to a first or fixed reference position for each parameter. The movements 
10 of die camera are sensed and die signals are fed into a combiner circuit 24 
and then to a transmit buffer 36 from which die combined video and 
position data signals are transmitted. 

During set up, at the receiver (Figure 7) the receive buffer 706 
15 receives the signals and feeds these to a splitter 708. The video signal is 
stored and delayed in a suitable store 710 and the camera parameter data 
is extracted and stored in store 712. 

In set up the VDU 700 is used to mark each billboard that may 
20 require replacement. The camera 20 is panned etc to move each billboard 
into a suitable position on the screen and its position is recorded in the 
billboard store 704 together with the camera parameters obtained from 
store 712 via processor 714. 

25 A replacement billboard store 716 stores a plurality of replacement 

billboards and these are selectable to be able to replace the original 
billboard. 
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The replacement billboard is in operation inserted into the video 
signal in a combiner 718 to provide a modified output video signal 720. 

16 



The setting up procedure can also identify billboard locations and 
camera parameters for several cameras by storing a camera ID from a 
source 30 (figure 6). Thus, billboard position store 704 will store separate 
lists of billboard data for each camera. 

5 

The operation of the system will now be described with reference 
to a single billboard and a single camera 20. 

With reference to Figure 4, it is assumed that billboard 14 enters 
10 the field of view in an enlarged form on the left-hand side of the screen 
as camera 20 pans following zooming from the Figure 3 position. 

The camera orientation data is constantly being received by the 
receiver and the processor 714 will constantly match on a pixel by pixel 

15 basis the video image with the known billboard position stored in store 
704. As soon as the billboard appears in the video image the pixels 
representing the billboard will be identified and the replacement billboard 
pixels which relate to those pixels will be substituted in the combiner 718. 
The delay will be minimal since the identification of the pixels is by an 

20 address correlation process which will be virtually instantaneous. 

After a period of time the camera sensors may drift and in this case 
the replacement billboard may not exactly align with the original. This 
may only be by one or two pixels and may not be discernible to the 
25 viewer. To correct this two solutions are possible. Firsdy, the billboard 
position can be manually restored periodically at a suitable time, for 
example, when a camera is not active. This requires the co-operation of 
the operator. 
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Secondly, a comparison can be made on a pixel by pixel basis of 

17 



the billboard against an original stored billboard and an adjustment of the 
reference camera parameters can be made in billboard position store 704. 
This process can be done automatically at either set intervals or when the 
processor 714 has a suitable time slot. 

5 

The essential steps of a preferred recalibration process are to 
perspectively transform the current video image using the camera data to 
provide an estimated transformed model. A stored image of the billboard 
is then compared with the transformed model to provide a residual video 
10 field. The residual distortion between the transformed model and the 
residual video field is resolved to provide updating information for 
updating the estimated transformation and to thereby provide a calibration 
correction factor for recalibrating the position of each billboard in the 
store in accordance with the camera sense information. 

15 

The replacement of each billboard is accomplished by use of the 
processor 714 (Fig. 7) and the various parameter and billboard stores 
using appropriate software programmes as now described in more detail. 

20 Figure 8 describes the complete process which allows to determine 

the position of each billboard in the camera's field of view, and render the 
corresponding part of the billboard into the frame buffer. Since the 
rendering and later the compositing of the graphics buffer with the video 
buffer by means of chroma-key are known art, we will concentrate on the 

25 billboard position determination with reference also to Figures 6 and 7. 

At the beginning of each video field, the pan, tilt, zoom and focus 
sensors (24,25,26) are read 800. These values, combined with billboard 
data from billboard setup data store 704 and camera data from camera 
30 intrinsic parameters store 712, enable the detection and recognition of all 
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billboards in the camera s FOV, independently of the video signal. The 
processing of Fig. 1 consists of a loop on all billboards (m) 802,804. For 
each billboard, its setup data is retrieved 806 from billboard setup data 
store 704 and used with camera intrinsic parameters 808 to compute the 
5 perspective transformation 810 from billboard m to current field. The 
replacement billboard information is then stored (812) in a frame buffer. 

Fig. 9 describes the billboard setup data store 900 which consists 
of a separate record 902.. 904 for each billboard in the arena. Such a 

10 record consists of a static image 906 grabbed in favourable conditions and 
of the corresponding static setup data 908. The record also consist of 
dynamic setup data 910 which is computed using the image processing 
means in a process known as dynamic re-calibration which has been 
briefly described above and will be further described with reference to 

15 Figure 11. 

Fig. 10 describes the setup data (either static or dynamic) 1000 for 
a single billboard. It consists of the sensors' readings 1002 at the setup 
instance, the billboard quadrilateral vertex 1004 coordinates and the time- 
20 code of the setup instance 1006. 

The method of dynamic re-calibration can be explained as follows: 

Due to sensors 9 drift and inaccuracies, a final calibration table and 
25 other practical reasons, it is impossible to predict the exact location of all 
visible billboards at a given instance. However, at many video fields, a 
billboard's visibility may be such that an exact geometric position 
correction can be performed. Since that position is closer both temporally 
and spatially to the subsequent video fields, it is preferable to rely on that 
30 "luck shot" by predicting the billboard position relative to its sensors' 
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readings ami exact quad coordinates. Consider for example a billboard 
which exits the field of view due to camera panning. Having a luck shot 
while it is still highly visible, allows the smooth tracking of the billboard 
by sensors only, when its visibility does not allow any image processing 
5 means to be applied. 

Fig. 11 depicts the flow diagram 1100 for perspective 
transformation computation. A setup data selection logic 1102 selects 
either the static 1103 or the dynamic 1105 setup data from the setup data 
10 store 806 as described above. This setup data, together with camera 
intrinsic parameter is used to compute a sensor-based prediction of the 
perspective transformation 1104, independently of the video signal. 

A dynamic re-calibration 1106 based on image processing means 
IS is then applied to the prediction. It utilizes the video 1108 and chroma- 
key 1110 signals as well as the billboards model image 1112 from the 
setup data store 806 (Fig. 8). Based on a quality factor derived from the 
image processing means, either the sensors-based 1118 or the corrected 
transformation 1 1 16 are output. If the estimated quality of the geometric 
20 correction is high, then the dynamic setup data is updated 1 1 14. 

Figs. 12,13,14 describe the sensor-based prediction of billboard 
coordinates in the video field. Such a prediction utilizes the sensors 
reading as well as the camera intrinsic parameters. These parameters are 
25 described in Fig. 12 and have to be tabulated for a dense sampling of the 
(zoom, focus) space. The meaning of these parameters is clear from 
Figure 13 to which reference is now made. 

Let the set of measurements given by the pan, tilt, zoom and focus 
sensors be represented by the vector (P,T,Z,F). The tilt angle is assumed 
30 to be relative to the horizon. 



20 



Consider an object point whose image at some setup instance is, at 
frame-buffer coordinates fe, y$). Let also the sensor measurement vector 
at that instance be (P s , T s , Zs, Fs). 

At another instance, the prediction instance, let the sensor 
measurements vector be (Pp, TV, Zp, F P ). It is required to predict the 
location of the object point in frame-buffer coordinates (possibly out of the 
actual frame-buffer), (x P , y P ). 

To enable the procedure we define the setup rotation matrix as 
shown at 600 and the prediction rotation matrix is defined as shown at 
602. 

Then, the perspective transformation matrix between the two image- 
plane coordinate systems is given as shown at 604 and 1402 (Fig. 14). 

Rsp is a 3*3 matrix with row and column indices ranging from 0 to 
2. RspPIQ] denotes the terra in row i, column j in the matrix. Thus, 
given the setup image-plane coordinates of the object point (Us, Vs), the 
predicted location of the object point in image-plane coordinates (up. v P ) 
is given as shown at 606, 1404. 

Image-plane to frame-buffer coordinate transformation is achieved 
as shown at 608, 1406. Abberration compensation is achieved as shown 
at 608, 1406 (Fig. 14) to provide predicted frame buffer billboard 
coordinates and perspective transformation data. 

An effective way of deriving these parameters for a specific 
(zoom,focus) pair is described in [J. Weng et al., Calibration of stereo 
cameras using a non-linear distortion model, IEEE 10th Intl. Conf. Pattern 
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Recognition (1990), pp. 246-253]. The image processing means for 
geometric correction, which allow also the process of re-calibration, is 
now described with reference to Fig. IS. 

5 The image processing means for geometric correction of sensors- 

based prediction are based on the differential method for motion estimation 
[C. Cafforio and F. Roca, The differential method for motion estimation, 
in: T.S. Huang, eg., Image sequence processing and dynamic scene 
analysis, Spring, Berlin, 1983, pp. 104-124]. Let C be the current video 
10 field and let M be the static billboard setup image, perspective^ 
transformed according to the sensors-based prediction. Here we consider 
only luminance images. Ideally, M and C are identical within the support 
of the billboard quadrilateral. Actual differences may include: 

• Occlusion present in C but not in M. 

IS • Geometric errors due to sensors and intrinsic camera 

parameters errors. 

• Luminance changes. 

Neglecting for the moment any difference which is not due to 
20 geometric errors, consider a point (x,y) inside the support of the billboard 
quadrilateral. Let (p,q) be the local geometric error then we may write 
for the luminance signals of the respective images: 
M(x + p,y + q) = C(x,y) 
Under the assumption that the error is small, one may write a 
25 Tailor series expansion: 

M{x + p,y + g) = M{x 9 y) + p ^ + q ^ + (Second order terms) 

dx dy 

Neglecting the second order terms and denoting the spatial 
derivatives 
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dM/dx = H 
dM/dy = V 

we obtain 

C(x,y)-M(x,y)=pH + qV 

5 

Also denoting the differences C(x,y) - M(x,y) by D we obtain 
D = pH + qV 

The equation above holds, locally. For a global billboard solution, 
10 and small error assumption we may use the perspective model [G. Adiv, 
Determining Three-Dimensional Motion and Structure from Optical Flow 
Generated by several moving objects, IEEE Trans. Pattern Analysis and 
Machine intelligence, 7, pp. 384 - 401, 1985]. 
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SjX + ay + 1 



The coefficients al,..,a8 are computed by minimizing the following 
expression: 



25 E Wxtf-p(x,yiKx 9 y)-<£xJ) Wx>?>? 



Now, the perspective transformation matrix (based on sensors 9 
30 prediction) is multiplied by: 
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5 The matrix obtained can be considered to be the updated prediction 

of billboard perspective. 

In a practical environment the following considerations may apply. 

• Occlusion may cause major problem in this formulation, 
since if pixels from occluding and moving objects participate 

10 in the minimization of the expression above they might bias 

the solution significantly. Preferably, such pixels are 
discarded from processing by using chroma-key panels. A 
key signal output by a chroma-keyer, is preferably utilized 
to discard these pixels. 

15 • Luminance variations can be minimized by pre-processing 

the current video field, using histogram matching 
techniques. 

• The prediction-correction process may require 2-3 iterations 
to converge. 

20 • Noise immunity and convergence can both be enhanced by 

pre-smoothing the images. 
Thus the billboards 14 etc are preferably chroma-key boards and 
occlusion is by colour discrimination using the normal chroma-key 
techniques. These techniques will enable perfect occlusion providing that 

25 the players do not wear any colour which is the same as the board. This 
may not always be possible and it is proposed in accordance with a 
particular embodiment of the present invention to use boards which can 
rotate or otherwise change to a second or third colour. For example, 
three colours may be blue, green and red which may be selected when the 

30 colour of the players strips are known. 
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In a preferred embodiment chroma-key apparatus can comprise the 
ULTIMATTE-7 digital video image compositing device from 
ULTTMATTE Corp., 20554 Plummer St., Chatsworth, CA 91311, USA. 

5 The backing colour can be selected between blue, green and red. 

In order for die chroma-keyer to calculate all parameters necessary to 
perform proper image compositing, the system requires a sample of the 
background colour as reference. This step can be done automatically by 
scanning the image and detecting the purest and brightest colour. 
10 Advanced chroma-keyers enable the user to manually select the area to be 
sampled. 

In a particular embodiment it is proposed to use a patterned 
chroma-key panel. Calibration of the camera sensors can then readily be 
15 accomplished by comparison of the pattern on a pixel by pixel basis. The 
pattern on the billboard panel should preferably have critical dimensions 
less than anticipated sensors error (projected to world coordinates). 

In summary, the above system can operate even in extremely poor 
20 weather conditions since the electronic processing circuitry knows exactly 
where each billboard is situated and does not rely on any analysis of the 
video image to detect the billboard. In the event that the video image is 
so distorted that recalibration cannot be carried out with reasonable 
certainty, then the original camera parameter settings can continue to be 
25 used since the video image as viewed will be of poor quality and thus the 
viewer will not notice an error of one or two pixels in the positioning of 
the replacement billboard which will require to be displayed in an 
equivalent quality which matches the poor quality video image. 

30 
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CLAIMS 

1 . Apparatus for automatic electronic replacement of a billboard in a 
video image including an automatic camera orientation measurement 
apparatus including motion measurement means operative to measure the 

5 Field of View (FOV) of the TV camera relative to a known reference 
position, 

2. Apparatus for automatic electronic replacement of a billboard in a 
video image as claimed in chum 1, including image processing means for 

10 processing video signals generated by the TV camera, in which said 
processing means includes calibration means for periodically automatically 
calibrating the motion measurement means. 

3. Apparatus as claimed in chum 2 in which the motion measurement 
IS means includes means for measuring the pan tilt, zoom or focus of the 

camera relative to known reference positions. 

4. Apparatus as claimed in claim 2 in which the motion measurement 
means includes means for measuring the pan tilt, zoom and focus of the 

20 camera relative to known reference positions. 

5. Apparatus as claimed in claim 2 further including a billboard in 
which said billboard comprises a chroma-key panel. 

25 6. Apparatus as claimed in claim 5 in which the panel comprises a 
patterned chroma-key panel. 

7. Apparatus as claimed in claim 6 in which the pattern comprises two 
different colours or two different shades of the same colour. 

30 
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8. Apparatus as claimed in claim 7 in which the pattern is designed to 
be optimised for use according to the size and shape of die billboard and 
to the anticipated video conditions. 

5 9. Apparatus as claimed in claim 2 in which the calibration means 
includes fine adjustment image processing means including means for 
perspectively transforming the current video image to provide an estimated 
transformed model, means for storing a replica of the billboard image, 
means for comparing the perspectively transformed video image with the 
10 stored replica to provide a residual video field, means for solving residual 
distortion between the transformed model and the residual video field and 
means for updating the estimated transformation by means of the residual 
distortion to provide a calibration correction factor. 

IS 10. Apparatus as claimed in claim 9 in which said means for storing the 
residual distortion includes analysis means based in the spatial derivatives 
of the current image of the transformed model as well as means for 
analysing pixel by pixel the image differences of the transformed model 
and the current image. 

20 

1 1 . Apparatus as claimed in claim 9 in which the fine adjustment means 
includes means for carrying out the fine adjustment in an interactive 
manner. 

25 12. Apparatus as claimed in claim 10 in which the pixel by pixel 
analysis means includes means for discarding pixels using colour variation. 

13. Apparatus as claimed in claim 2 including means for recording 
calibration data obtained during a set up period when no occlusion is 
30 present. 
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14. Apparatus for automatic measurement of camera orientation 
including means for measuring the Field of View (FOV) of the camera 
relative to a known reference position, and including image processing 
means for processing the video signals generated by the camera, said 

5 image processing means including calibration means for periodically 
calibrating the motion measurement means. 

15. A method of automatic replacement of a billboard in a video image 
said method comprising the steps of: 

10 recording the Field of View (FOV) of a TV camera in a first 

reference position; 

storing the image viewed by the TV camera in said first reference 

position; 

measuring the movement of the camera relative to the first 
IS reference position and recording the measurements and periodically 
automatically calibrating the motion measurements by comparison of the 
present video image produced by said TV camera with the stored video 
image in said reference position; 

calibrating a difference signal for recalibration of the motion 
20 measurements and said first reference position and replacing a real 
billboard in said cameras field of view with a virtual replacement billboard 
in accordance with said recalibrated first reference position. 

16. A method as claimed in claim 15 in which said step of measuring 
25 the movement of the camera relative to the first reference position 

comprises the steps of measuring the pan tilt, zoom or focus of the 
camera. 

17. A method as claimed in claim 15 in which said step of measuring 
30 the movement of the camera relative to the first reference position 
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comprises the steps of measuring the pan tilt, zoom and focus of die 
camera. 

18. A method as claimed in any one of claims 15 to 17 in which the 
5 billboard comprises a chroma-key panel and in which said step of 
replacement of the real billboard by the virtual billboard includes the step 
of occluding said replacement billboard by any object occluding said real 
billboard. 

10 19. A method as claimed in any one of claims IS to 18 in which said 
step of periodically automatically calibrating the motion measurements 
includes a fine adjustment image processing step including perspectively 
transforming the current video image to provide an estimated transformed 
model, storing a replica of the billboard image, comparing the 

15 perspectively transformed video image with the stored replica to provide 
a residual video field, solving residual distortion between the transformed 
model and the residual video field and updating the estimated 
transformation by means of the residual distortion to provide a calibration 
correction factor. 

20 

20. A method as claimed in claim 19 in which said step of storing the 
residual distortion includes analysis based in the spatial derivatives of the 
current image of the transformed model and analysing pixel by pixel the 
image differences of the transformed model and the current image. 

25 

21. A method as claimed in claim 19 in which the fine adjustment step 
comprises carrying out the fine adjustment in an interactive manner. 

22. A method as claimed in claim 20 in which the pixel by pixel 
30 analysis step includes discarding pixels using colour variation. 
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23. A method as claimed in claim IS in which said step of recording 
calibration data is carried out during a set up period when no occlusion is 
present. 



5 24. A method as claimed in claim 15 in which said step of calibrating 
a difference signal is carried out only when the present video image is 
above a predetermined quality factor. 

25. A method as claimed in claim 24 in which said virtual replacement 
10 billboard is positioned at the last recalibrated position in the event that a 
recalibration is not possible due to the present video image being below 
the predetermined quality factor. 
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